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ABSTRACT 
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engage in constructing personally meaningful products, that is meaningful to 
themselves or to others around them. The construction of knowledge from 
experiences in the world seems to be especially important in childhood 
learning, as children need to learn through their senses and through physical 
activity. Unfortunately, for school children there are few learning 
situations where senses and physical activity is involved. This is partly due 
to the nature of the concepts to be learned and partly to the lack of 
manipulative learning material. This paper introduces a mixed reality 
environment as a collaborative and constructive learning space for elementary 
school children. During recent teaching experiments in an elementary school 
the children created their own mixed reality environment which served from a 
scientific point of view for a semiotic-based understanding for the creation 
and usage of physical and digital media bringing together arts and computer 
science. Includes five color figures. (Contains 13 references.) (Author) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



■CR021774 ED 477 046 



PERMISSION TO REPRODUCE AND 
DISSEMINATE THIS MATERIAL HAS 
BEEN GRANTED BY 






TO THE EDUCATIONAL RESOURCES 
INFORMATION CENTER (ERIC) 



Mixed Reality Environments 
as Collaborative and Constructive Learning Spaces 
for Elementary School Children 



Huberta Kritzenberger, Thomas Winkler, Michael Herczeg 



Institute for Multimedia and Interactive Systems 
University of Luebeck 
Willy-Brandt-Allee 31a 
D-23554 Luebeck, Germany 

{kritzenberger, winkler, herczeg] @imis .mu-luebeck.de] 



uTTice oi Educational Research and Improvement 

educational resources INFORMATION 

/ CENTER (ERIC) 

^This document has been reproduced as 
' received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 



Abstract. Learning is an active, constructive and collaborative process, where people construct knowledge from their 
experiences in the world. People construct new knowledge with particular effectiveness when they engage in constructing 
personally meaningful products, that is meaningful to themselves or to others around them. The construction of knowledge 
from experiences in the world seems to be especially important in childhood learning, as children need to learn through their 
senses and through physical activity. Unfortunately, for school children there are few learning situations where senses and 
physical activity is involved. This is partly due to the nature of the concepts to be learned and partly to the lack of 
manipulative learning material. This contribution introduces a mixed reality environment as a collaborative and constructive 
learning space for elementary school children. During recent teaching experiments in an dementary school the children 
created their own mixed reality environment which served from a scientific point of view for a semiotic-based understanding 
for the creation and usage of physical and digital media bringing together arts and computer science. 



1 Introduction 

Education in the media age means to develop media competence and a deeper understanding of media characteristics starting from 
kindergarten age to high school education. The learning goal is to understand the design as well as the use of the media and also to 
understand the essence and nature of specific media. 

Starting from these learning goals the authors are involved in a project called ArtDeCom (’Theory and Practice of Integrating Education and 
Training in Arts and Computer Science”). We tiy bringing together education in the two usually separated disciplines of arts and computer 
science in school on several levels of education. Pupils should be given an opportunity to explore the relationships of the two disciplines 
with adequate tools and learning goals for their respective age and learner level. The project is funded by the German “Bund-Lander 
Commission for Educational Planning and Research Promotion” (BLK) within the general funding program "Culture in the Media Age". It 
aims at investigating teaching material and situation for a curriculum definition for integrating education in arts and computer science on all 
educational levels. The IMIS is cooperating with the Muthesius Academy of Arts, Design and Architecture and with the Institute of Art 
History of the Christian- Albrechts-University in Kiel (Germany) to develop curricular elements for an integrated education of arts and 
computer science. 

This situation is the background for the integrated lessons of arts and computer science in an elementary school in Luebeck (Germany). The 
teaching attempt is in the 3“^^* class level of an elementary school for children with an age of 8-9. During several lessons the children work in 
design projects developing their own mixed reality environment, within which they later perform an interactive musical revue. The children 
themselves develop the scenes of the mixed reality environment working in groups and using adequate software tools as well as real world 
materials. Collaborative and constructive learning is practiced permanently during the design process as well as during interaction with the 
hybrid environments of the mixed reality scenery. 

A key focus of our research refers to cross-settings of ways of thinking in arts and in computer science. We think that mixed reality 
environments are a perfect learning space that enables children to explore concepts from both disciplines in an intuitive way. 

2 Collaborative and Constructive Learning 

Constructive learning theory addresses learning as an active process, in which people actively construct knowledge from their experiences in 
the world. People construct new knowledge with particular effectiveness when they engage in constructing personally meaningful products, 
which are meaningful to themselves or to others around them. Furthermore, it is important that the learning environment is authentic and 
situated in a real-life situation. Learners must get an opportunity to build multiple contexts and perspectives in a social context. Numerous 
examples have been given in academic context. 
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The construction of knowledge from experiences in the world seems to be especially important in childhood learning, as children need to 
learn through their senses and through physical activity. Todays kindergartens and playgrounds are full of physical objects and physical 
activity. But as children move on through elementary school and into secondary school, they encounter fewer learning situations where 
senses and physical activity is involved. This is partly due to the nature of the concepts to be learned. They are very difficult to explore 
with the senses when manipulative learning material is not available. One reason is that many abstract concepts seem to be very difficult (if 
not impossible) to explore with physical media. For example, traditional physical media generally do not support children understanding the 
behavior of dynamic systems or how patterns arise through dynamic interactions among component parts. Such concepts are typically 
taught through more formal methods, involving abstract mathematical formalisms. Unfortunately, many students have severe problems with 
this approach, and thus never develop a deep understanding of these concepts (Resnick 1998). 

With this background in mind Resnick (Resnick 1998) has created a new generation of computationally enhanced manipulative materials, 
called "digital manipulatives", developed at the MIT Media Lab. They expand the range of concepts that children (and adults) can explore 
through direct manipulation of physical objects. They aim to enable children to continue to learn concepts with "kindergarten approach" 
even as they grow older. As children build and experiment with these manipulative materials, they form mental models and develop deeper 
understanding of the concepts they enact with. Children continue to learn new concepts with a "kindergarten approach". Resnick assumes 
that children learn with digital manipulatives concepts that were previously considered "too advanced" for them. 

There are several other approaches with physical and tangible (graspable and touchable objects) interaction, for example MIT's KidsRoom 
(Bobick et al. 2000), Triangles and 'strings’ (Gorbet et al. 1998), Curlybot (Frei et al. 1999); StoryMat (Ryokai/Cassel 1999) in order to 
enhance collaboration among learners and enable constructive learning experiences for children. 

Furthermore, several design projects with children have shown positive effects on learning and deep understanding (Alborzi et al. 2000; 
Druin/Perlin 1994; Stanton et al. 2001). In these projects children create external artifacts, like animate stories, video games, kinetic 
sculptures, models, simulations and so on, which they share and discuss with others. These artifacts provide rich opportunities for learning. 
As children are involved as active participants, they have a greater sense of control over the learning process. As they design artifacts in 
group work, they experience pluralistic thinking, multiple strategies and solutions. By the way they need to think about how other people 
will understand and use their constructions. Furthermore, design projects are mostly interdisciplinary and therefore bring together concepts 
from different disciplines. 

3 Mixed Reality Environments as Learning Spaces 

In our approach to constructive learning we use a mixed reality environment as a collaborative and constructive learning space for children. 
As we have discussed in the previous chapter, in early elementary classrooms children need manipulative material to initiate learning 
processes, as they allow exploration with the senses and therefore enable physical experience for the learner. They allow children to explore 
concepts by interaction and with their senses and therefore allow to build and to correct mental models and a deeper understanding of the 
underlying domain. 

In mixed reality environments real world and virtual world objects are presented together on a single display. Mixed Reality techniques have 
proven valuable in single user applications. For example in other contexts single user mixed reality interfaces have been developed for 
computer aided instruction (Feiner/Mcintyre/Seligmann 1993) and for medical visualization (Bajura/Fuchs/Ohbuchi 1992). These 
applications have shown that mixed reality interfaces can enable a person to interact with the real world in ways, which were never possible 
before. However, there has been less research on collaborative applications (Billinghurst/Kato 1999) or on interaction of children with mixed 
reality environments (www.animax.de) or even multi-user mixed reality environments and their use as learning spaces. 

3.1 Design of Mixed Reality Environments 

The mixed reality environments used in our lessons are designed by the children themselves in collaborative design projects. In the beginning 
the class is divided into g’oups, where each group works out a project development plan. It includes a story, the material to be used, the 
function of the computer and its connections and extensions to physical space. Afterwards there is a phase of creating, collecting and 
arranging objects for the mixed reality environment. These objects can either be of digital or of physical nature. Various materials for scenery 
creation are used, like wires and papier-mache (see figure 1). During the development process the pupils are acquainted with the hardware 
and software to be used. 

Children collaboratively create their own mixed reality environment. During this working process the children get involved as active 
participants creating something meaningful to themselves and to others. 
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Figure 1: Children are creating the objects for the mixed reality scene 

Furthermore, during the working process the children do not only create an animation for the digital world, but they also create the digital 
world. In our project they simply use a LEGO Cam for that task and LEGO MindStorms Vision Command to program the interactive 
sequences of certain picture recognition sequences (see figure 2). 




Figure 2: Children producing an animation with the LEGO Cam 



3.2 Scenario: The ’’World of Dragons" music revue in a mixed reality environment 

Figure 3 shows a sketch of the collaborative mixed reality environment, which was designed (as described in Chapter 3.1) by the children for 
a music revue "World of Dragons". It consists of an animation, which shows a wild landscape with mountains, trees, a watercourse, an 
active volcano, a cave and several moving dragons. On the stage within the physical world, there are several trees which have been made 
from the same material and look exactly like the trees in the animation. Between the projection wall and the video animation the children 
move and dance on the stage. They wear differently colored dragon costumes and dance to the music of Sergey Proko^ew ("Romeo and 
Juliet", "The Montagnes and Capulets"). The rhythm of the music as well as the narrative structures of the animation impose a certain 
timing of the acoustic and body expression. 
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Figure 3: Sketch of the collaborative mixed reality environment for the music revue 

By their movements and by the colors of their costumes the children activate different sounds in the mixed reality environment. There are 
also other sounds coming from objects in the mixed reality environment, like the blowing of the wind, the rolling of thunder, the volcanic 
eruption, the sound of an avalanche, the shouting of dragons, as well as the recurring sound of a stumbling little dragon. 

3.3 How children learn from mixed reality environments 

The learning experience of the children can be interpreted on several levels. On a concrete level of interpretation, children learn about 
concepts and methods in both disciplines, in arts and in computer science (see figure 4). 
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Figure 4: Connecting methods of arts and of computer science 



In an active hybrid environment the activities of the children ground in a physical world. They are engaged in the manual development of the 
"landscape". They create and realize the objects, which are used for the animation. They act and dance on the stage and interact with the 
animation by their movements and they influence the behavior of the computer program with signs like the color of their costumes. With 
iconic programming the learners bridge the gap between the physical world with its multiple meanings and interpretations and the abstract 
world of models and algorithms in computer science. Furthermore, an adequate tool is needed to realize programming experience for 
children. For our purposes we found LEGO MindStorms, Robotic Invention System and Vision Command fitting perfectly as a 
programming environment for the purposes of our project. 

The most interesting aspect of children's learning experiences in mixed reality environments is of semiotic nature and can be explained with 
the "teridentity relation" (x=y=z) of Charles Sanders Peirce. 
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Figure 5: Terldentity relation for the interpretation of learning experiences In mixed reality environments (acccording to 



Whereas in traditional ways of teaching only separated areas are interpreted from a semiotic point of view, in our form of transdisciplinary 
teaching we try to relate all possible signs to each other. It does not matter, whether these signs are of iconic nature, or if they are indices, 
symbols, individual events and states, or habits of predictive / relational, propositional or argumentative nature. 

If we interpret the learning experiences of the children in a more systematic and semiotic way, we get the following situation and 
interpretations. According to Peirce (HartshomeAVeiss/Burks 1997) there are 10 possible kinds of signs, where Peirce endeavored to 
construct a theory of all possible natural and conventional signs, whether simple or complex. This theory is helpful to explain how different 
levels of interpretation and learning experience are related to each other. We do not want to classify the elements of the transdisciplinary 
learning environment according to these 1 0 classes of Pierce, because this conclusion would neglect a lot and shift the focus to the theory. 
Instead we would like to focus to very important practical aspects: 

■ experience in design and creation of the narrative story; the dragons mirror the fascination and the anger of the world of the dragons as 
well as the kind of relationships between the children and their appraisals 

■ experience of the relationship to the body, to corporeality and being touched by others, especially children of the other sex 

■ experience of elements of mimic and dancing expressions (which exists independently of verbal expressions) 

■ empathy in music and rhythm, with its attributes like loud and soft, fast and slow, pretty and grave 

■ experience of the quality of material during the creation of the landscape (which is either light or heavy, hard or soft, dry or slippery) 
and examination of the shaping of the three dimensions of the room 

■ experience of the pictorial nature and modelling process during the creation of the digital animation, when single picture sequences are 
constructed from gradually changing the real world models 

■ identification of characteristic sounds and their integration into the narrative and expressive dance story of the music revue 

■ experience in abstract information processing for conception and development of the picture recognition program with LEGO-RIS 
Vision Command 

■ reflection on the different relationships of boys and girls to dancing as well as to computer technology 

■ experience on how attribute of objects in the real world (like colour, movement, position in the real world space) cause reactions in the 
virtual world, that is on the states and effects of the computer program 

4 Conclusions 

In our teaching attempt we found that mixed reality environments are a very promising learning space for a constructive and collaborative 
learning of children. The mixed reality environment meets all requirements for a constructive learning space for children, like learning through 
senses and physical activity using digital manipulatives. 

In designing and in working with a mixed reality environment, the children construct a configuration of the world instead of a describing 
knowledge. They do this by individually creating and constructing the world and by subsequently freely interpreting their self-created 
world. During the performance of the music revue the children experience a singular event. However, for every event numerous 
interpretations are possible. By interpreting the world the children do not create an analytical but an intuitive understanding. The children 
are engaged in the world of constructing the objects, in the world of programming and in the world of behavior. They write programs and 
they control the computer program with their behavior (dancing, movements, colors of their costumes etc.). By this the children become 
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themselves variables in the computer program when acting in the world of the musical revue. They act themselves as signs with different 
states and effects on each other as well as on the elements of the real and the virtual world. 

Further research will explore the use of other computer-based learning environments as tools and environments for trans-disciplinary 
teaching and understanding in the field of arts and computer sciences. The research will be done on different levels of education and age. 
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